Istituto Di Matematica Computazionale

The size of electronic data is currently growing at a faster rate than computer memory and disk storage capacities. For this reason compression appears always as an attractive choice, if not mandatory. However space overhead is not the only resource to be optimized when managing large data collections; in fact data turn out to be useful only when properly indexed to support search operations that efficiently extract the user-requested information. Approaches to combine compression and indexing techniques are nowadays receiving more and more attention. A first step towards the design of a compressed full-text index achieving guaranteed performance in the worst case has been recently done in [10]. The novelty of that index resides in the careful combination of the compression algorithm proposed by Burrows and Wheeler [6] with the suffix array data structure [16]. The index is opportunistic in that, although no assumption on a particular fixed distribution is made, it takes advantage of the compressibility of the input data by decreasing the space occupancy at no significant asymptotic slowdown in the query performance. In this paper we present an implementation of this index and perform an extensive set of experiments on various text collections. These experiments allow us to highlight properties and drawbacks of the proposed solution, as well as identify some interesting scenarios where this novel index may find effective application. ∗Dipartimento di Informatica, Università di Pisa, Italy. E-mail: [email protected]. Supported in part by Italian MURST project “Algorithms for Large Data Sets: Science and Engineering” and by UNESCO grant UVO-ROSTE 875.631.9. †Dipartimento di Scienze e Tecnologie Avanzate, Università del Piemonte Orientale, Alessandria, Italy and IMC-CNR, Pisa, Italy. E-mail: [email protected]. Supported in part by MURST 60% funds.